---
title: scraper
date: 2021-08-10 09:34:18
tags: python, scraping, web
---

I just wrote a web-scraping tool based on [Scrapy](https://docs.scrapy.org).
There was a lot of new things that poped up during the build that I want to register while it's still fresh in my mind.  
My idea was this,
- check what are the steps necessary to have a spider up and running,  
- Turn all steps into python functions,  
- Create a class to host them. It facilitates the communication between the functions,  
- Add the needed information as class instances items.  
In the end it worked pretty well. The build was uneventful and I did what I set out to do. Could it be always like that.  
The app is made from two modules, one that houses the class and does all the heavy lifting, the other, a lighter, simpler file, where you input data to build the instance that'll become your scraper. I also call the functions from this file.  
Regarding the class, the file is called 'scraper.py', and it's built like this:  
In the \__init__ function of the class I define what will be the instances requisites.
The items are:  
- Name: Name of the publication.
- Project Name: Name for the project folder. Personally I just use 'name' and prefix it with '_info'.  
- Path: Path to your project.  
- Beginning URLs: The pages that you want to scrape first.  
- Content Title: The title that accompanies the news.  
- Content Author: Who wrote the piece.  
- Content Date Altered: When, if when, was the piece revised.  
- Content: The text of the piece.  

```python
   def __init__(self, name, project_name, path, start_urls, title, author, date_altered, content):
        self.name = name
        self.project_name = project_name
        self.path = path
        self.start_urls = start_urls
        self.title = title
        self.author = author
        self.date_altered = date_altered
        self.content = content
```
First we have go to where we want to keep the projects. 'os.chdir' is a Python command to change the current working directory. os.mkdir creates a directory.  
```python
    def dislocation1(self):
        """We head for designated local for the app and create a folder to house it"""
        os.chdir("/home/mic/python/scraper")
        os.mkdir(f"{self.project_name}")
```
The next step is to let Scrapy create the project folders and files. For that we are going to use a command line command, which forces us to run it through Subprocess. The 'cwd=path' part, instructs Subprocess to execute the command in the specified folder.  
```python
    def start_project(self):
        """Scrapy command to start a project"""
        path = f"/home/mic/python/scraper/{self.project_name}"
        cmd = "scrapy startproject " + self.name
        subprocess.run(cmd, cwd=path, shell=True)
```
It is now time to start building the spider. For that it's needed to run another command-line, command. Because of this, the structure of the function is very similar to the former.  
```python
    def start_spider(self):
        """Scrapy command to initiate a spider"""
        project_name = self.project_name
        name = self.name
        paths = "/home/mic/python/scraper/" + project_name + "/" + name + "/" + name + "/spiders"
        cmd = f"scrapy genspider {self.name}_info {self.start_urls}"  # O nome tem de ser diferente do projecto.
        subprocess.run(cmd, cwd=paths, shell=True)
```
We now need to erase some text from the files and add a new one. In this function will do only the erasing. We will erase the string "http", because when we put our urls,it latches on to them and creates URL's with this format: "http://https://new_url.com".  
There's also a class that expects our input, so now it only has written "pass" on it. We'll erase it. Because this implies changing a file, we need to take the following steps:  
1. First we change current working directory to where the files are, with os.chdir.  
2. We open the original file in 'read' mode, and create an output file to put the original text plus the changes we want to make.  
3. We replace "http://" with nothing.  
4. We replace "pass" with nothing.  
5. We write this changes on the output file.  
6. We remove the original file.  
7. We change the output file name to the name of the original file.  
 ```python
    def edit_spider_file_clean_file(self):
        """Here we clean the spider file a bit. It comes a bit dirty when it is generated by Scrapy"""
        os.chdir(f"/{self.path}/{self.project_name}/{self.name}/{self.name}/spiders")
        with open(f"{self.name}_info.py", "r") as infile, open(f"{self.name}_output.py", "w") as outfile:
            data = infile.read()
            data = data.replace("http://", "")
            data = data.replace("pass", "")
            outfile.write(data)
            os.remove(f"{self.name}_info.py")
            cmd = f"mv {self.name}_output.py {self.name}_info.py"
            subprocess.run(cmd, shell=True)
 ```
 Then we arrive to the trickiest function to write. We have to insert strings on a file, taking in account that they need to be in particular places on it. This was harder than I thought.  
 The first four changes pertain to [Xpath](https://www.w3.org/TR/xpath20/) commands, that'll define what from where in the webpage we'll be scraping.  
 They all take the form of variables and all have the suffix "response.xpath()", that indicates that we're seeing something collected from the scraping, the "response" part, encoded in Xpath. The browser's developer tools lets us choose a CSS element and copy its Xpath address, making the job even easier than using css names. They will all be organized as a Python dictionary with the [zip](https://www.w3schools.com/python/ref_func_zip.asp) command. If its not a dictionary, XML, JSON and some other type, Scrapy won't accept it.  
 We open the file in writing and execute the commands therein.  
```python
 def edit_spider_file_parse_function(self):
        """We add to the file information regarding the areas of the site we want to target"""
        os.chdir(f"/{self.path}/{self.project_name}/{self.name}/{self.name}/spiders")
        with open(f"{self.name}_info.py", "a") as f:
            f.write(f"        title = response.xpath('{self.title}').extract()" "\n")
            f.write(f"        author = response.xpath('{self.author}').extract()" "\n")
            f.write(f"        date_altered = response.xpath('{self.date_altered}').extract()" "\n")
            f.write(f"        content = response.xpath('{self.content}').getall()" "\n")
            f.write("        data = zip(title, author, date_altered, content)")
            f.write("\n")
            f.write("        for item in data:\n")
            f.write(
                "            info = {'title': i:wtem[0], 'author': item[1], 'date_altered': item[2], 'content': item[3]}"
            )
            f.write("\n")
            f.write("        yield info")
            f.write("\n")
            f.write("        with open('content.txt', 'w') as cont:")
            f.write("\n")
            f.write(f"            for i in response.xpath('{self.content}').getall():")
            f.write("\n")
            f.write("                cont.write(str(i))")
```
We now change folders with os.chdir(), to get to the settings.py file, and add two settings:  
1. We define the feed format. 'json' in this case.  
2. The name of the file that will host the scraped information.  
```python
    def settings(self):
        """We add information regarding the format of the output"""
        os.chdir(f"{self.path}/{self.project_name}/{self.name}/{self.name}")
        with open("settings.py", "a") as f:
            f.write('FEED_FORMAT = "json"')
            f.write("\n")
            f.write(f"FEED_URI = '{self.name}.json'")
        print("Changes to settings are done")
```
Now we start, finally, crawling the site. As it's another command line command, we'll be using Subprocess again.  
```python
    def crawl(self):
        """We initiate the scraping"""
        path = self.path
        project_name = self.project_name
        name = self.name
        path = path + "/" + project_name + "/" + name + "/" + name + "/spiders/"
        cmd = "scrapy crawl " + name + "_info"
        subprocess.run(cmd, cwd=path, shell=True)
```
We now open the file created on the last step, and check if all was scraped correctly.  
```python
    def result(self):
        """Verifying the output file, to see if everythng is ok"""
        name = self.name
        path = f"{self.path}/{self.project_name}/{self.name}/{self.name}/spiders/"
        cmd = "vim " + name + ".json"
        subprocess.run(cmd, cwd=path, shell=True)
```
Here is all the code:  
```python
"""The objective of this module is to automate the steps to scrape a given publication"""
import os
import subprocess
import scrapy
from loguru import logger

fmt = "{time} - {name} - {level} - {message}"
logger.add("spam.log", level="DEBUG", format=fmt)
logger.add("error.log", level="ERROR", format=fmt)


class Scraper:
    """This class will be broken in the steps needed to create a scraping campaign. Each step a function."""

    def __init__(self, name, project_name, path, start_urls, title, author, date_altered, content):
        self.name = name
        self.project_name = project_name
        self.path = path
        self.start_urls = start_urls
        self.title = title
        self.author = author
        self.date_altered = date_altered
        self.content = content

    def dislocation1(self):
        """We head for designated local for the app and create a folder to house it"""
        os.chdir("/home/mic/python/scraper")
        os.mkdir(f"{self.project_name}")

    def start_project(self):
        """Scrapy command to start a project"""
        path = f"/home/mic/python/scraper/{self.project_name}"
        cmd = "scrapy startproject " + self.name
        subprocess.run(cmd, cwd=path, shell=True)

    def start_spider(self):
        """Scrapy command to initiate a spider"""
        project_name = self.project_name
        name = self.name
        paths = "/home/mic/python/scraper/" + project_name + "/" + name + "/" + name + "/spiders"
        cmd = f"scrapy genspider {self.name}_info {self.start_urls}"  # O nome tem de ser diferente do projecto.
        subprocess.run(cmd, cwd=paths, shell=True)

    def edit_spider_file_clean_file(self):
        """Here we clean the spider file a bit. It comes a bit dirty when it is generated by Scrapy"""
        os.chdir(f"/{self.path}/{self.project_name}/{self.name}/{self.name}/spiders")
        with open(f"{self.name}_info.py", "r") as infile, open(f"{self.name}_output.py", "w") as outfile:
            data = infile.read()
            data = data.replace("http://", "")
            data = data.replace("pass", "")
            outfile.write(data)
            os.remove(f"{self.name}_info.py")
            cmd = f"mv {self.name}_output.py {self.name}_info.py"
            subprocess.run(cmd, shell=True)

    def edit_spider_file_parse_function(self):
        """We add to the file information regarding the areas of the site we want to target"""
        os.chdir(f"/{self.path}/{self.project_name}/{self.name}/{self.name}/spiders")
        with open(f"{self.name}_info.py", "a") as f:
            f.write(f"        title = response.xpath('{self.title}').extract()" "\n")
            f.write(f"        author = response.xpath('{self.author}').extract()" "\n")
            f.write(f"        date_altered = response.xpath('{self.date_altered}').extract()" "\n")
            f.write(f"        content = response.xpath('{self.content}').getall()" "\n")
            f.write("        data = zip(title, author, date_altered, content)")
            f.write("\n")
            f.write("        for item in data:\n")
            f.write(
                "            info = {'title': i:wtem[0], 'author': item[1], 'date_altered': item[2], 'content': item[3]}"
            )
            f.write("\n")
            f.write("        yield info")
            f.write("\n")
            f.write("        with open('content.txt', 'w') as cont:")
            f.write("\n")
            f.write(f"            for i in response.xpath('{self.content}').getall():")
            f.write("\n")
            f.write("                cont.write(str(i))")

    def settings(self):
        """We add information regarding the format of the output"""
        os.chdir(f"{self.path}/{self.project_name}/{self.name}/{self.name}")
        with open("settings.py", "a") as f:
            f.write('FEED_FORMAT = "json"')
            f.write("\n")
            f.write(f"FEED_URI = '{self.name}.json'")
        print("Changes to settings are done")

    def crawl(self):
        """We initiate the scraping"""
        path = self.path
        project_name = self.project_name
        name = self.name
        path = path + "/" + project_name + "/" + name + "/" + name + "/spiders/"
        cmd = "scrapy crawl " + name + "_info"
        subprocess.run(cmd, cwd=path, shell=True)

    def result(self):
        """Verifying the output file, to see if everythng is ok"""
        name = self.name
        path = f"{self.path}/{self.project_name}/{self.name}/{self.name}/spiders/"
        cmd = "vim " + name + ".json"
        subprocess.run(cmd, cwd=path, shell=True)
```

##### The Main Function
On another file named 'main_scraper.py', we define the instances arguments for the class that houses the functions we just saw.  
Then the last thing we have to do is run sequentially the functions that were created, to have a simple and fast spider bot functioning and running smoothly.  
```python
#!/usr/bin/python3.9
"""Main module of the app. Where all functionalities are accessed from"""
import os
import scrapy
import questionary
from scraper import Scraper
from loguru import logger

fmt = "{time} - {name} - {level} - {message}"
logger.add("spam.log", level="DEBUG", format=fmt)
logger.add("error.log", level="ERROR", format=fmt)


def main():
    """Here we start, sequentially, all the steps needed to create a scraping campaign."""
    rasoura = Scraper(
        "eco",
        "eco_newspaper",
        "/home/mic/python/scraper",
        "https://eco.sapo.pt/2021/08/10/chineses-obrigam-loja-da-xiaomi-em-portugal-a-desistir-de-pagamentos-com-criptomoedas/",
        '//*[@id="post-878113"]/div/div[1]/header/h1/text()',
        '//*[@id="post-878113"]/div/div[1]/header/div[2]/div/div[1]/ul/li[1]/a/text()',
        '//*[@id="post-878113"]/div/div[1]/header/div[2]/div/div[1]/ul/li[2]',
        '//*[@id="post-878113"]/div/div[1]/div[1]/p/text()',
    )

    rasoura.dislocation1()
    rasoura.start_project()
    rasoura.start_spider()
    rasoura.edit_spider_file_clean_file()
    rasoura.edit_spider_file_parse_function()
    rasoura.settings()
    rasoura.crawl()
    rasoura.result()


main()
```
